Resolved -
Between 14:30 - 17:18 UTC, we experienced delays in error ingestion due to some components of our ingestion pipeline adding significant load to our primary database. Average error ingestion delay reached a maximum of 22.5 minutes at around 15:23 UTC.
The issue was resolved and error ingestion is operating as expected.
May 28, 19:37 UTC
Monitoring -
Our ingestion backlog has recovered and everything looks good currently. We're continuing to investigate improvements to our ingestion pipeline to prevent similar bottlenecks in the future. Average event ingestion latency is normal at around 15 seconds.
May 28, 17:28 UTC
Identified -
We've identified an issue causing bottlenecks within our event ingestion pipeline. We are working on optimizing a few areas of our pipeline and are catching up on our ingestion backlog. Average event ingestion delay is now under 8 minutes and continuing to drop.
We apologize for any inconvenience caused by this and expect it to be resolved in the next 1 hour. If anything changes, we'll provide another update.
May 28, 16:54 UTC
Update -
We're continuing to investigate this issue and will provide another update as soon as we've identified the root cause. Maximum error delays peaked at around 22.5 minutes and are currently just under 19 minutes.
May 28, 15:55 UTC
Investigating -
We're currently investigating reports of a delay in errors ingestion and will provide further updates as soon as we have more information.
May 28, 15:05 UTC